Then you become busy with others leaving him.
You ignored him
of him you were unmindful
To him thou payest no regard
So you are from him distracted/preoccupied
the who is mindful of his Lord
So you leave him, and are engrossed elsewhere
Him shouldst thou neglect
But you are not focusing your attention on him
But you neglect him and pay him no regard
Of him you are neglectful and divert your attention to another